Python Web Scraping, Second Edition by Packt Publishing

Author:Packt Publishing
Language: eng
Format: mobi
Publisher: Packt Publishing
Published: 2017-05-29T09:14:54+00:00

def process_queue():

while len(crawl_queue):

url = crawl_queue.pop()

...

The first change is replacing our Python list with the new Redis-based queue, named RedisQueue. This queue handles duplicate URLs internally, so the seen variable is no longer required. Finally, the RedisQueue len method is called to determine if there are still URLs in the queue. Further logic changes to handle the depth and seen functionality are shown here:

## inside process_queue

if no_robots or rp.can_fetch(user_agent, url):

depth = crawl_queue.get_depth(url) or 0

if depth == max_depth:

print('Skipping %s due to depth' % url)

continue

html = D(url, num_retries=num_retries)

if not html:

continue

if scraper_callback:

links = scraper_callback(url, html) or []

else:

links = []

# filter for links matching our regular expression

for link in get_links(html, link_regex) + links:

if 'http' not in link:

link = clean_link(url, domain, link)

crawl_queue.push(link)

crawl_queue.set_depth(link, depth + 1)

The full code can be seen at http://github.com/kjam/wswp/blob/master/code/chp4/threaded_crawler_with_queue.py.

This updated version of the threaded crawler can then be started using multiple processes with this snippet:

import multiprocessing

Download

Copyright Disclaimer:
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.